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NUCLEAR SMUGGLING DETECTION: RECENT 
TESTS OF ADVANCED SPECTROSCOPIC POR- 
TAL MONITORS 


Wednesday, March 5, 2008 

U.S. House of Representatives, 

Committee on Homeland Security, 
Subcommittee on Emerging Threats, Cybersecurity, and 

Science and Technology, 

Washington, DC. 

The subcommittee met, pursuant to call, at 2:10 p.m., in Room 
311, Longworth House Office Building, Hon. James R. Langevin 
[chairman of the subcommittee] presiding. 

Present: Representatives Langevin, Christensen, Green, Pascrell, 
and McCaul. 

Mr. Langevin. The subcommittee will come to order. 

The subcommittee is meeting today to receive testimony on two 
recent reports: The Independent Review Panel Report and DNDO’s 
Phase III Test Report which details a test of advanced 
spectroscopic portal monitors. 

Before I go into the hearing itself and start with my opening 
statement, we are expecting votes around 2:15, unfortunately. It is 
my intention to try to get as far through the opening statements 
as possible; then we will recess for what is I understand one vote. 
Then we will come back and continue with statements if necessary, 
and then go into questions. 

Good afternoon. I want to thank the witnesses for being here at 
this very important hearing. 

Today we are discussing a very important project for the Domes- 
tic Nuclear Detection Office, and that is the advanced spectroscopic 
portal monitor program. 

This subcommittee held its first hearing on this topic 1 year ago. 
We will continue to provide robust oversight on this project until 
we are assured that we are deploying the best technology possible 
to detect radiological and nuclear materials coming across our bor- 
ders. 

Given that we are holding a public hearing on this topic, I think 
it is fair to say that we are not quite there yet, though I do com- 
mend DNDO for improving our screening capabilities, along with 
both our southern and northern borders as well as our seaports. 

We are currently scanning 100 percent of all incoming cargo on 
the southern border, 98 percent at the Nation’s seaports, 98 per- 
cent at the northern border. Mr. Oxford assures me that we will 
be at 100 percent on the northern border by next year. I think 
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these are important points of progress, and I want to acknowledge 
that we are making progress in these areas. 

While I applaud DNDO for its aggressive pursuit of new detec- 
tion technologies, I still remain deeply concerned that the advanced 
spectroscopic portal monitors have not been properly tested and 
evaluated. In one of the reports that we will discuss today, the 
independent review team reiterates a recommendation made by 
GAO over a year ago: Deployment and testing should not be done 
by the same organization. This lack of rigorous and independent 
testing program can easily lead to the development and even the 
deployment of ineffective equipment. 

The two reports under consideration today both raise as many 
questions and concerns as they answer. It is my hope that the ad- 
vanced spectroscopic portal monitors will ultimately function as in- 
tended, but at this point I expect Secretary Chertoff will require 
many more unequivocal test results before certifying to Congress 
that the ASP represents a significant increase in operational effec- 
tiveness over the currently deployed systems. 

Perhaps the field test underway by GBP will add to this needed 
level of clarity and if not, all the steps, I know, will. 

Not all the news presented in these reports is bad. There are sev- 
eral results that point to progress of the ASP program. However, 
both reports are extremely nuanced and do not seem to give clear, 
strong indications of whether this project is achieving its stated 
goals. In fact, there are several statements included in these re- 
ports that, if taken at face value and on their own, would cause 
most people to say that ASP has failed or, at the very least, is in 
serious jeopardy of doing so. 

For example. Phase III report states that when it comes to iden- 
tifying mass sources, the polyvinyl toluene performed — PVT sys- 
tems perform better than ASP. However, upon a more extensive re- 
view of the entire report, it becomes evident that this statement 
does not mean exactly what it seems to say, and I will explore this 
issue further in my questions. 

I don’t want to see this program fail; none of us do. Let me say 
this again: I don’t want to see the program fail; it is far too impor- 
tant. The advantages of an effective spectroscopic portal program 
along the current binary alarm/no alarm option under the current 
PVT monitors would represent a major increase in Homeland Secu- 
rity, but only if they work as intended. 

Current tests will still have not unequivocally demonstrated in 
an operational setting that the ASP represents a significant im- 
provement over current technologies. 

This is an important point. While it may be easy to only focus 
on the simplistic big-picture items, this is a subtle and nuanced 
issue that we cannot afford to overlook. I hope that this hearing 
will allow us to delve into the details of these two reports and to 
make clear exactly what was said and what was not and where we 
go from here. 

I should note that the GAO has been a great help and a trusted 
source on this issue for Congress for various reasons. No GAO rep- 
resentatives were able to attend today’s hearing for legitimate rea- 
sons, but we will continue to rely on their counsel in the future and 
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implore those witnesses here today to cooperate with GAO and to 
fulfill their role as Congress’ trusted auditor. 

With that, I just want to again thank the witnesses for being 
here today and look forward to your testimony. 

The Chair now recognizes the Ranking Member of the sub- 
committee, the gentleman from Texas, Mr. McCaul, for an opening 
statement. 

Mr. McCaul. I thank the Chairman. I agree with your assess- 
ment that we do not want to see this fail. The American people 
can’t afford to see it fail. It is too important. 

I want to begin by thanking the witnesses for being here. Mr. 
Oxford, you are certainly no stranger to this committee. At this 
time last year you were here to describe DNDO’s deployment strat- 
egy for radiation portal monitors. As we saw in your written testi- 
mony, you have assisted Customs and Border Protection, I should 
say, in deploying monitors that are now scanning incoming cargo 
for radiological and nuclear materials at a volume of 91 percent on 
our northern border and 98 percent at our Nation’s seaports. Com- 
ing from a border State myself, I was very pleased to see the 100 
percent screening on our southern, southwest border. 

I commend you and your staff for your hard work and service to 
the Nation. I am sure the Department will agree that while 
progress has been made in scanning incoming cargo, the current 
generation of radiation portal monitors is far from perfect. Fre- 
quent nuisance alarms due to radioactive, but legitimate material 
such as cat litter or medical therapeutics require manpower-inten- 
sive secondary screening by Customs officers. It is very time-con- 
suming but not as efficient as new technology hopefully will pro- 
vide. 

To improve the efficiency of scanning people and cargo at our 
ports of entry, DNDO has led a development program to provide 
the next generation of radiation portal monitors, detectors that can 
discriminate between threat materials that can be used in a nu- 
clear weapon or dirty bomb and other materials that pose no threat 
to the Nation at all. The advanced spectroscopic portal monitor also 
known as ASP could provide this capability. 

I commend the Chairman on holding this hearing today as part 
of our continued oversight of the ASP program, a program that re- 
sides solely within the jurisdiction of this committee. 

It has been said that we have to get it right all the time, while 
the terrorists only have to get it right once. However, if we can es- 
tablish a system of layered defenses, that would turn that argu- 
ment on its head; now the terrorist has to get it right at each layer 
in the system, and we only have to get it right once. 

I see the ASP program as a critical layer in our system of defense 
against the transport of radiological and nuclear threat material. 
The delays in the certification and deployment of the ASP system, 
a technology that could accurately identify threat materials with- 
out impeding commerce, are of grave concern to this committee. 

That is why today Ranking Member King of the full committee 
and I have introduced legislation that will assist the Department 
in certifying this technology by clarifying congressional intent on 
the requirements for certification and the metrics to be considered 
in evaluating ASP’s system performance. This legislation should 
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help the Department keep their intended goal of making a decision 
on ASP certification by the end of this fiscal year. 

I hope we can achieve that goal within the end of this fiscal year, 
and I thank the witnesses for being here. 

Thank you, Mr. Chairman. 

Mr. Langevin. I thank the Ranking Member. Given the fact that 
the vote has just been called, we will recess right now for about 10 
minutes for what I believe is just one vote and then return for the 
witnesses’ statements. 

The committee stands in recess. 

[Recess.] 

Mr. Langevin. The committee will come to order. The Ranking 
Member has been detained, but he has given his indication to go 
ahead without him, and he will return momentarily. 

I want to welcome, again, our first panel of witnesses. Our first 
witness is Mr. Vayl Oxford, Director of the Domestic Nuclear De- 
tection Office at the Department of Homeland Security. He has ap- 
peared before this panel many, many times to discuss this and 
other topics. We welcome him back here today. 

The next witness is Dr. George Thompson, who is the Deputy Di- 
rector of Programs at the Homeland Security Institute. After sev- 
eral other individuals stepped down from the position of Chair of 
the Independent Review Panel, Dr. Thompson took up the position, 
which allowed the independent review team to complete its work. 
Welcome. 

Our third witness is Ms. Elaine Duke, the Deputy Under Sec- 
retary for Management at the Department of Homeland Security. 
Her office originally called for an independent review panel to be 
done, and her office will be a key part of incorporating the findings 
of the panel’s report into its recommendations to the Secretary on 
whether to ultimately certify the ASP program for full procure- 
ment. 

So I want to thank the witnesses for being here. Without objec- 
tion, the witnesses’ full statements will be inserted into the record. 

I now ask each witness to summarize his or her statement for 
5 minutes, beginning with Secretary Duke. 

STATEMENT OF ELAINE C. DUKE, DEPUTY UNDER SECRETARY 
FOR MANAGEMENT, DEPARTMENT OF HOMELAND SECURITY 

Ms. Duke. Good afternoon. Thank you, Mr. Chairman, Ranking 
Member McCaul, and members of the committee. It is a pleasure 
to appear before you today to talk about the advanced spectroscopic 
portal system. Today is my first time before you as the Deputy 
Under Secretary for Management. I have been in this position for 
about 5 months but have spent most of my 25 years of public serv- 
ice in the procurement profession, most recently as the Depart- 
ment’s chief procurement officer. 

The Deputy Under Secretary of Management position was cre- 
ated as a part of the Department’s transition planning efforts to 
ensure operational continuity during the change of administration 
in January 2009. My position currently holds the authority of the 
Under Secretary for Management, as the current Under Secretary, 
Mr. Paul Schneider, is serving as the Acting Deputy Secretary. 
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To start I would like to convey my top priorities which are essen- 
tial elements to achieving the DHS mission and practicing sound 
stewardship of the taxpayers’ money. 

First is preparing the Department for the first administration 
transition; second, improving acquisition and procurement; and 
third, strengthening the requirements process and integrating it 
into the planning, programming, budgeting and execution process 
in the Department. 

My goal as the Deputy Under Secretary for Management as it re- 
lates to transition is to focus on three areas: internal processes, 
knowledge management and relationship building. 

In addition to transition planning and focusing on transforming 
the procurement office into a full-fledged acquisition office, often 
procurement and acquisition are incorrectly used interchangeably. 
Procurement is just one element of the acquisition management. 
Today we are talking about another one, test and evaluation. 

Acquisition includes the full operation life cycle requirements 
process using sound business strategies, financial management and 
managing program risks. We are making progress toward this goal. 
In August 2007 we established the Acquisition Program Manage- 
ment Division to provide oversight and support the acquisition pro- 
grams. 

Today we have performed assessments of 37 of our largest level 
one programs and have provided advice and guidance, particularly 
in the area of cost/benefit analysis. To that end, I am here today 
to discuss one of our major acquisition programs, the ASP program. 

The acquisition of ASP systems is of national importance and 
vital priority for the Department to continue toward this mission 
of protecting the country from dangerous goods. The acquisition de- 
velops the next-generation radiation portal monitor and has the 
ability to rapidly identify the presence and type of radioactive ma- 
terials present in the cargo entering the United States. It will allow 
us to distinguish from harmless sources, thereby decreasing the 
rate of false alarms resulting in unhindered flow of commerce. 

Before DHS uses its appropriated funds to deploy this new tech- 
nology, Congress has directed Secretary Chertoff to certify that a 
significant increase in operational effectiveness will be achieved. In 
July 2007, the Secretary announced his intent to perform an inde- 
pendent review of the ASP test procedures, test results and associ- 
ated technology assessments. 

My role as the Deputy Under Secretary in this process is to un- 
derstand the outcome of this independent review and be in a posi- 
tion to advise the Secretary in making his certification decision. In 
my opinion, as DHS considers the best way forward, this inde- 
pendent review provides valuable assistance to the Secretary and 
me in moving toward conclusion of this program. 

The independent review is not an unusual exercise as it is in line 
with reviews we have conducted in other major programs. It is a 
standard practice we are developing, modeled after the Department 
of Defense, and will help us in our decisionmaking process and our 
acquisition programs. 

There are several things we have already learned from the inde- 
pendent review. First, to define up-front increased operational ef- 
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fectiveness so we can appropriately test ASP’s capabilities, which 
is critical to our ability to certify the system. 

Second, to develop a new test and evaluation master plan that 
will clearly demonstrate the test results and whether ASP does in- 
deed provide the increased operational effectiveness. 

Third, to include all the ownership costs in updating the cost/ 
benefit analysis for this new phase of the program. 

I have asked Mr. John Higbee to lead this effort from an acquisi- 
tion perspective. He is the director of our Acquisition Program 
Manager Provision; and Mr. George Ryan, from our Under Sec- 
retary of Science of Technology, who is our director of Test and 
Evaluation Standards. They provide valuable insight as we move 
forward in the next test phase of this program. 

Mr. Chairman, I thank you for the opportunity to testify before 
you and am happy to answer any of yours and the committee’s 
questions. Thank you. 

Mr. Langevin. Thank you, Ms. Duke, I appreciate your testi- 
mony. 

[The statement of Ms. Duke follows:] 

Prepared Statement of Elaine C. Duke 
March 5, 2008 

Thank you Mr. Chairman, Representative McCaul and Members of the committee. 
It is a pleasure to appear before you today to talk about the Advanced Spectroscopic 
Portal (ASP). 

This is my first time before you as the Deputy Under Secretary for Management 
(DUSM). I have been in this position for over 5 months but have spent most of my 
25 years of public service in the procurement profession, most recently as the De- 
partment’s Chief Procurement Officer. 

The Deputy Under Secretary for Management position was created as part of the 
Department’s 2009 Administration Transition Planning efforts. By having a senior 
career civil servant in this capacity, rather than a political appointee, the Depart- 
ment can ensure operational continuity during the change in administration. As the 
current Under Secretary for Management, Mr. Paul Schneider is serving as the Act- 
ing Deputy Secretary, my position holds the authorities of the Under Secretary for 
Management. 

At present, the most significant management challenge the Department has is 
continuing the effort that was mandated at the Department’s creation: merging 22 
agencies with approximately 208,000 people and turning it into the most effective 
force to protect our country. This effort requires effective and efficient use of finan- 
cial and human resources; enabling technology, strong processes and superb man- 
agement. It is toward this effort that I devote my time, energy, and contributions. 

As the Deputy Under Secretary for Management, it is my duty to lead the Man- 
agement Directorate’s efforts in the development of the Department, with a focused, 
well-thought strategy. 

The major elements of our strategy are: 

• Improving acquisition and procurement throughout the Department. 

• Strengthening the requirements and investment review processes. 

• Acquiring and maintaining human capital. 

• Seeking efficiencies across the enterprise in operations and the use of resources. 

• Making the key management systems, such as financial and information tech- 
nology, world class. 

• Acquire funding for DHS’ consolidation at St. Elizabeths West Campus and the 
efficient realignment of all Department of Homeland Security (DHS) off-campus 
locations 

To start, I would like to convey my top priorities, which are essential elements 
to achieving the DHS mission and practicing sound stewardship of taxpayers’ 
money: 

• First: Preparing for the Department’s first ever administration transition; 

• Second: Improving acquisition and procurement; 
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• Third: Strengthening the requirements process and integrating it into the Plan- 
ning, Programming, Budgeting, and Execution (PPBE). 

My goal as the DUSM as it relates to Transition is to focus on three areas: Inter- 
nal Processes, Knowledge Management, and Relationship Building. The Internal 
Processes initiative will review our Directives, strengthen records management and 
our processes for incoming and exiting employees. The Knowledge Management ini- 
tiative will produce briefing materials, but more importantly, convey to career ex- 
ecutives and incoming appointees the requisite knowledge to keep the Department 
running. The Relationship Building initiative will facilitate direct interactions 
among Federal, State, local and tribal officials with homeland security responsi- 
bility. 

In addition to transition planning, my focus is to transform the Office of Chief 
Procurement Officer (CPO) into an Acquisition Office. Often, Procurement and Ac- 
quisition are incorrectly used interchangeably. Procurement, however, is only one 
element of acquisition management. Acquisition also includes understanding oper- 
ational and life-cycle requirements, such as formulating concepts of operations, de- 
veloping sound business strategies, exercising prudent financial management, as- 
sessing tradeoffs, and managing program risks. Best practice acquisition manage- 
ment is executed by teams of professionals who understand and are able to manage 
the entire life-cycle of a major program effort. We are making progress toward this 
goal. 

The Acquisition Program Management Division (APMD) of CPO began operations 
in August 2007. The division was established to provide oversight and support for 
acquisition programs. To date APMD has performed Quick Look assessments of 37 
level 1 programs and has overseen Deep Dive reviews of the SBInet and ASP pro- 
grams. APMD has provided advice and guidance to a number of programs, particu- 
larly in the area of cost benefit analysis. Currently the APMD team is focused on 
an aggressive Investment & Acquisition process re-engineering effort. The effort in- 
cludes replacing Directive 1400, establishing revised investment and acquisition de- 
cision procedures, as well as processes for, acquisition program baselining, periodic 
reporting, acquisition of services, and other initiatives. 

DHS’ $17 billion procurement spend plan provides for the development, fielding 
and support of significant homeland security capabilities. For example, U.S. Coast 
Guard contracts are providing aircraft and ships from the Integrated Deepwater 
System (IDS) and search and rescue capability from the Rescue 21 program. Trans- 
portation Security Administration (TSA) contracts are providing additional capabili- 
ties via the Electronic Baggage Screening Program (EBSP). Consistent with the SBI 
Strategy, U.S. Customs and Border Protection (CBP) is developing and fielding the 
capabilities at and between our Nation’s ports of entry to gain effective control of 
our borders. The Domestic Nuclear Detection Office is developing and testing a new 
type of radiation portal monitor known as the Advanced Spectroscopic Portal (ASP) 
to improve the Nation’s defense against the threat of nuclear smuggling. 

I am here today to discuss the Advanced Spectroscopic Portal (ASP). The acquisi- 
tion of ASP systems is of national importance and vital priority for the Department 
to continue toward its mission of protecting the country from dangerous goods. This 
acquisition develops the next generation Radiation Portal Monitor (RPM) and has 
the ability to not only detect the presence of radiation in cargo entering the United 
States, but also to rapidly identify the type of radioactive material(s) present. 

In 2005, the Domestic Nuclear Detection Office (DNDO) took on the responsibility 
to develop a second generation RPM prototype, now known as ASP. The intent of 
this initiative was to decrease the rate of false alarms and close the gaps in cov- 
erage. Or in other words, increase operational effectiveness and rapidly detect the 
presence and the type of radioactive material present. This system would allow us 
to distinguish harmless sources, such as kitty litter from those that might pose a 
threat. As a result of these improved detection capabilities, the flow of commerce 
would proceed unhindered. 

Before Congress would appropriate the funds to deploy this new technology, it in- 
cluded restrictive language in the Homeland Security Appropriations Act for Fiscal 
Year 2007 requiring the Secretary to certify that a significant increase in oper- 
ational effectiveness will be achieved before funds will be appropriated. In July 
2007, Secretary Chertoff announced his intent to “assemble a highly experienced 
team” to perform “an independent review of the [ASP] test procedures, test results, 
[and] associated technology assessments”. 

My role as the DUSM in this process is to understand the outcome of this inde- 
pendent review and be in a position to advise the Secretary in determining whether 
he should certify that there will be a significant increase in operational effectiveness 
with the procurement of ASP systems. In my opinion, this independent review pro- 
vides valuable assistance to the Secretary, to the Department Acquisition Executive, 
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Chair of the DHS Investment Review Board and me, as DHS considers the hest way 
forward. 

This is not an unusual exercise as it is in line with the reviews we conduct for 
our all our major programs within the Department. Furthermore, this is a standard 
practice of other Departments within the U.S. Government, such as Defense, to im- 
prove their decisionmaking processes regarding major programs. 

The Department appreciates the need for rigorous review to ensure that the De- 
partment acquires the crucial capability to preventing the smuggling of nuclear ma- 
terials across our borders. It is entirely appropriate for DHS to leverage the re- 
sources of the executive branch to gather information to make an informed decision 
on a critical program. We consider the independent review of this system to be com- 
plementary to GAO’s investigation of ASP. As an agent of Congress, GAO provides 
information to Congress in support of its oversight function. We intend to review 
and consider these reports from both sources in determining a way forward. 

There are several things we have already learned from the Independent Review. 
First, ASP operational testing is critical to our ability to certify the system. I have 
asked Mr. John Higbee, Director, Acquisition Program Management to oversee ASP 
operational testing, working with independent Operational Test and Evaluation ex- 
perts both internal and external to the Department. By conducting ASP operational 
testing, we will improve our ability to make an informed decision on this program. 
Testing of this type will also allow us to continue to exercise more oversight over 
the Department’s acquisition programs as well as strengthen our requirements and 
investment review process. 

Second: We are mindful of the need for the ASP program to demonstrate “in- 
creased operational effectiveness”, and the interest Congress has in this criterion. 
We are working with DNDO and CBP to ensure that the ASP testing program is 
structured to conclusively determine this critical point. 

Third: We learned the “cost” portion of the Cost Benefit Analysis (CBA) should 
include all ownership costs, including maintenance and support; and the “benefits” 
portion of the CBA should be consistent with the logic used to define “operational 
effectiveness.” Therefore, we will carefully review the CBA to ensure that the alter- 
natives are well-defined and that the assumptions, data inputs, and calculations are 
sound. 

Finally: After reviewing and addressing the Independent Review Team’s, and 
GAO’s findings; and after considering the test results and all information provided 
within this process, the Secretary will be prepared to make a decision on whether 
he should certify that there will be a significant increase in operational effectiveness 
with the procurement of ASP systems. 

Mr. Chairman, thank you for the opportunity to testify before the committee on 
this very important topic. I would be glad to answer any questions you or the Mem- 
bers of the committee may have for me. 

Mr. Langevin. The Chair now recognizes Mr. Oxford for 5 min- 
utes. Welcome. 

STATEMENT OF VAYL S. OXFORD, DIRECTOR, DOMESTIC NU- 
CLEAR DETECTION OFFICE, DEPARTMENT OF HOMELAND 

SECURITY 

Mr. Oxford. Good afternoon, Mr. Chairman, Ranking Member 
McCaul and other Members of the subcommittee. I would like to 
thank the subcommittee for the opportunity to discuss recent 
progress in ASP development and the recently released Phase III 
test report and the final report of the ASP independent review 
team. 

Before addressing these reports, I would like to update the com- 
mittee on progress since I last appeared before you. Since that time 
as has been previously mentioned, we have reached several mile- 
stones. 

First of all, in December we met the congressionally mandated 
goal of scanning 98 percent of the cargo coming in through our sea- 
ports. This is significant improvement over where we were 3 years 
ago when we were only scanning 22 percent of that cargo. When 
we couple that with 100 percent of the cargo we are scanning 
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across the southern border, we are now scanning 96 percent of all 
cargo entering the United States on a daily basis. This is real and 
measurable progress. In working with CBP, we have a plan to com- 
plete the northern border in 2009. 

Also in December we completed delivery of radiation detection 
equipment for all U.S. Coast Guard boarding teams. To address 
other threat pathways, we also delivered additional hand-held de- 
tection equipment to Customs and Border Protection; and as of De- 
cember, Customs and Border Protection is now scanning all inter- 
national general aviation airplanes arriving in the United States. 

Finally, DNDO has developed plans and is in the early stages of 
implementing a program of interest to this subcommittee to en- 
hance physical security of high-risk radioactive sources in U.S. 
medical facilities. 

With respect to the IRT report I would like to highlight several 
key points: 

First it is important to recognize that DNDO is developing and 
evaluating detection systems in response to established threat 
guidance originally established by the Department of Energy for 
Customs and Border Protection. This approach establishes require- 
ments to detect and identify quantities of plutonium or uranium 
that are actually a fraction of the likely amount or the amount like- 
ly required by a terrorist to construct nuclear weapons. 

Second and most importantly, the IRT reports that it could not 
find any signs of biased testing or manipulation of data. We thank 
the IRT for its efforts and intend to adopt several aspects of the 
recommendations. However, we find some shortfalls in the report. 

DNDO and CBP believe that ASP’s systems provide significant 
advantages in both primary and secondary scanning roles, and 
there is a possibility that the greatest benefit will actually be real- 
ized in primary applications. 

The IRT report focused on secondary applications and as a re- 
sult, did not include data showing that current rpms may miss 
some critical threats resulting in threats never being referred to 
secondary applications. 

By not conducting a full system, the system comparison, the ASP 
benefit of reducing the number of secondary referrals, was also ig- 
nored. Based on current data, ASP systems have the potential to 
reduce secondary referral rates by a factor of 5 to 10 resulting in 
up to 150,000 less secondary inspections a year. 

For secondary applications, ASP systems provide a more con- 
sistent scan, as acknowledged by the IRT, and avoid the localiza- 
tion issues associated with using a hand-held detector in secondary 
scanning applications. As a result of being able to quickly scan the 
entire container and using time slicing to sample regions of interest 
and reduce background, ASP systems will be significantly more ef- 
ficient and effective than the current hand-held device. 

Regarding the Phase III report. Phase III testing served many 
purposes, which provided an initial assessment of the range of ASP 
capabilities. As such, the tests subjected ASP systems to a variety 
of sources, masking cases and shielding configurations, and in- 
cluded scenarios more challenging than threat guidance that I pre- 
viously mentioned. 
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For example, we tested five shielding thicknesses that varied in 
difficulty, and above and below threat guidance level. We tested six 
types of masking materials of a representative cargo entering the 
United States, as well as challenging combinations involving indus- 
trial and medical sources that may potentially mask the presence 
of a threat. 

The bottom line is that we have explored ASP performance 
around the threat guidance to determine the extent of current ASP 
performance and enable algorithm improvement. We are using this 
data in the current system upgrades. 

Going forward, UNDO and CBP are planning additional tests to 
verify these capabilities leading to secretarial certification later 
this fiscal year. 

In the interest of time, Mr. Chairman, I will use a chart showing 
the upcoming test series that are available to you on the screens 
during the question-and-answer period if that is okay. 

With that, I will conclude my testimony and be glad to answer 
any questions. 

Mr. Langevin. Thank you. Director Oxford. 

[The statement of Mr. Oxford follows:] 

Prepared Statement of Vayl S. Oxford 
March 5, 2008 

INTRODUCTION 

Good morning, Chairman Langevin, Ranking Member McCaul, and distinguished 
members of the subcommittee. As Director of the Domestic Nuclear Detection Office 
(DNDO), my office is responsible for developing new technologies and also ensuring 
that we deploy detection systems properly across the domestic nuclear detection ar- 
chitecture. I would like to thank the committee for the opportunity to discuss recent 
progress in the development of the next generation of radiation portal monitors 
(RPMs), or Advanced Spectroscopic Portal (ASP) systems. My testimony today will 
focus principally on the recently released Phase III Test Report, the Final Report 
of the ASP Independent Review Team (ASP-IRT), and the steps we will take to 
make a certification and production recommendation to the Secretary. 

HISTORIC CONTEXT 

Countering the threat of nuclear terrorism is one of the top priorities for the De- 
partment of Homeland Security (DHS). DNDO is the lead agency responsible for the 
development, acquisition, and deployment of radiation detection equipment to sup- 
port this mission within the Department. The ASP program is one of the programs 
DNDO has begun to improve radiation detection tools for operators, in this case 
Customs and Border Protection (CBP) Officers. 

Considerable progress has already been made using currently available tech- 
nology. At ports of entry (POEs), RPMs are typically installed in a primary scanning 
location to detect the presence of radiation in cargo and vehicles. CBP operates addi- 
tional RPMs and handheld radioisotopic identification devices (RIIDs) in secondary 
scanning locations to further investigate alarms originating in primary and identify 
the specific source of the radiation detected. As of February 8, 2008, 100 percent 
of all incoming cargo on the southern border is being scanned for the presence of 
radiological or nuclear material, as well as 98 percent at the Nation’s seaports, and 
91 percent on the northern border. However, much work remains to close enduring 
gaps at many small border crossings along the northern border, as well as at small 
seaports. In addition, limits in the capabilities of current systems continue to 
present technical and operational challenges to those using the equipment. 

Unlike current systems which detect and identify radiation sources in a two-step 
process, ASP technology uses the radiation spectrum from the inspected material to 
make a single detection and identification decision. DNDO has maintained that this 
ability to differentiate between threat material and naturally occurring radioactive 
material (NORM) will reduce the number of alarms due to non-threat sources, re- 
duce the number of containers and vehicles sent to secondary inspection, and dra- 
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matically improve the probability of correctly identifying and interdicting smuggled 
nuclear material during secondary inspections. 

In 2006, the Congress requested that DNDO complete a cost-benefit analysis of 
ASP systems, which DNDO subsequently issued in June 2006. In a later report 
(GAO 07-133R), the Government Accountability Office (GAO) raised concerns about 
performance and cost assumptions included in the DNDO cost-benefit analysis, and 
the Congress included further restrictions in the fiscal year 2007 Appropriations Act 
(Pub. L. 109-295), directing that, “none of the funds appropriated under this head- 
ing shall be obligated for full scale procurement of [ASP] monitors until the Sec- 
retary of Homeland Security has certified . . . that a significant increase in oper- 
ational effectiveness will be achieved.” 

In order to provide the Secretary with all necessary information prior to a certifi- 
cation decision, DNDO launched a substantial test campaign from February 2007 
through September 2007. This included three separate test series conducted at the 
Nevada Test Site (NTS), including the Phase III testing captured in the report that 
we are discussing today, as well as contractor verification testing, stream of com- 
merce testing at the New York Container Terminal (NYCT), integration testing at 
the Pacific Northwest National Laboratory (PNNL), and field validation at eight 
operational sites. In addition, in late July 2007, Secretary Chertoff notified the Con- 
gress of his intent to “assemble a highly experienced team” to perform “an inde- 
pendent review of the [ASP] test procedures, test results, [and] associated tech- 
nology assessments.” This group, known as the ASP-IRT, delivered a report to 
Elaine Duke, the DHS Deputy Under Secretary for Management, on February 20, 
2008. The ASP-IRT Report is the second document under discussion today. 

PHASE III TEST REPORT — INTRODUCTION 

The Phase III Test Campaign, conducted at NTS in March 2007, was part of a 
larger series of tests conducted throughout 2007, designed to evaluate ASP perform- 
ance. Specifically, Phase III testing was intended to collect data from challenging 
detection or identification cases, beyond those included in Phase I testing at NTS 
earlier in the year. In addition, Phase III testing was to support the development 
of concepts of operations, and provide an additional data collection opportunity for 
continued vendor development of improved detection and identification algorithms. 
Phase III testing was conducted in accordance with test plans developed by DNDO, 
in partnership with the National Institute of Standards and Technology (NIST), 
CBP and, to a limited extent, the Department of Energy (DOE) and its labs. The 
test plan included the incorporation of a variety of source and shielding configura- 
tions, and, in particular, configurations that were often more difficult than “guid- 
ance” detection goals. This point is particularly important when analyzing the re- 
sults of the test. 


PHASE HI TEST REPORT — OBJECTIVES AND RESULTS 

The Phase III Test Campaign was designed to evaluate several aspects of ASP 
system performance, with five primary objectives. Before discussing the objectives 
and results, it is necessary to provide several caveats that relate to the way the 
tests were designed, and how the specific objectives of the test affected the interpre- 
tation of the results obtained. Eirst, it is important to reiterate that results indi- 
cating that ASP systems did not detect or identify some specific cases do not indi- 
cate that ASP systems did not work as designed. ASP systems are designed to oper- 
ate to certain design thresholds. In some instances. Phase III test sources inten- 
tionally exceeded those thresholds to evaluate how far ASP performance continues. 
For instance, a number of test sources were selected that were shielded beyond 
amounts identified in government requirements, for the express purpose of under- 
standing where ASP capabilities begin to “fall off.” The fact that ASP systems func- 
tioned beyond specified requirements should be considered a positive sign, rather 
than a sign of inherent flaws. 

Second, Phase III tests were intended to help DNDO and CBP better understand 
the full range of ASP system performance, and results will continue to guide further 
development efforts. Since testing in early 2007, DNDO has provided results as 
feedback to the ASP vendors, and they have incorporated this data into subsequent 
design improvements. These improvements will be evaluated through additional test 
campaigns scheduled for this year. 

Finally, detailed Phase III results are classified at the SECRET level. Because in- 
dividual results reveal vulnerabilities to both systems that are and will be deployed, 
performance of systems against specific sources cannot be discussed in an open set- 
ting. The results that follow have been intentionally generalized to avoid discussion 
of specific performance capabilities for systems. DNDO has previously provided clas- 
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sified test results to the committee staff. DNDO would be happy to provide the same 
information to committee Members in an appropriate environment. 

Detection sensitivity for plutonium surrogate. — The detection sensitivity of ASP 
systems was measured against a plutonium “surrogate,” or a source that was de- 
signed to mimic the detectable signature of plutonium. This objective was specifi- 
cally focused on ensuring that ASP systems met the CBP requirement that they be 
at least as sensitive as current-generation polyvinyl toluene (PVT)-based systems. 
ASP detection sensitivities were measured against PVT-based systems set at exist- 
ing operational thresholds. 

For this representative source, ASP systems were more sensitive than PVT-based 
systems when operating at existing operational thresholds. As such, testing met the 
objective of assessing that ASP systems did not degrade detection performance, as 
compared to current systems. 

Relative performance as a function of source categories. — Phase III testing sought 
to compare relative performance of various ASP systems as a function of source cat- 
egories. Source categories included bare, shielded, and “masked” special nuclear ma- 
terials, bare, shielded and “masked” industrial sources, and medical isotopes. In par- 
ticular, this objective sought to identify any significant variation in performance be- 
tween each ASP vendor design. Additionally, detection performance was also com- 
pared to PVT-based systems, and identification performance was compared to cur- 
rent-generation sodium iodide (Nal)-based handheld detectors that are currently 
used to conduct secondary inspections. 

Due to the number of source categories evaluated as part of this objective, results 
are more complicated than those associated with other objectives. While on average 
no ASP system significantly outperformed the others with regard to detecting 
sources passing through portals at either 5 or 2 miles per hour, there were dif- 
ferences in system performance when evaluated against each source. 

With regard to comparisons between PVT-based and ASP systems, ASP systems 
outperformed PVT-based systems in detecting bare special nuclear materials, and 
both types of systems performed similarly against shielded special nuclear mate- 
rials. “Masked” special nuclear materials resulted in higher alarm rates for PVT- 
based systems than ASP systems. Similarly, PVT-based systems demonstrated high- 
er alarm rates for medical isotopes and industrial sources, though this was due to 
ASP decision software that categorized smaller sources as “non-threats.” Based on 
requests from CBP, revisions have since been made to ASP algorithms so that in- 
dustrial sources will be referred for secondary scanning. Finally, and significantly, 
ASP systems outperformed handheld RIIDs in identifying all source categories, with 
the exception of bare industrial sources. Due to the extremely high signal strengths 
associated with industrial sources, performance between ASP systems and RIIDs 
was comparable in that instance. 

Effects of shielding on system performance. — This test campaign sought to provide 
preliminary measurements of the effects of shielding materials on system perform- 
ance. In particular, tests evaluated the difference in ASP system response when dif- 
ferent types of sources, including special nuclear materials, were placed inside vary- 
ing amounts of shielding. For the purposes of this objective, system “response” in- 
cluded both detection and identification performance. 

As expected, all systems experienced difficulty in detecting and identifying certain 
heavily shielded materials, which results in signal strengths significantly below cur- 
rent “guidance” levels and requirements. This is consistent with performance of all 
passive detection systems. However, ASP systems were able to identify sources 
when placed inside almost all but the thickest shielding configuration tested. 

Relative performance for combined sources. — Phase III testing evaluated the rel- 
ative performance of ASP systems against “combined sources,” where more than one 
emitting isotope was present. This portion of Phase III testing was designed to pro- 
vide additional data collection opportunities for ASP vendors, in support of algo- 
rithm improvements. This testing was not designed to provide conclusive data as 
to the performance of ASP systems against “masked” sources. 

Phase III testing highlighted several areas where further study and algorithm de- 
velopment are required to reduce vulnerabilities. This data was provided to the ASP 
vendors, and software improvements are being incorporated into ASP revisions. In 
addition, the use of high-purity germanium-based systems, when operated in a 
“wait-in mode,” showed slightly better performance than other systems. However, 
these initial results are an indicator of potential capabilities, rather than proof of 
superior performance. 

Secondary screening for concepts of operations development . — Additional evalua- 
tions were completed to assess varying concepts of operations for secondary scan- 
ning. ASP system performance was evaluated as a function of varying speeds and 
“dwell times,” or the amount of time that a source was present within the portal. 
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Specifically, measurements were conducted as sources moved at several pre-set 
speeds through the portals, as well as instances where sources were stopped within 
the portal for a defined amount of time. 

The evaluations of concepts of operations demonstrated that scanning at 2 miles 
per hour, the current concept of operations for secondary scanning, could be suffi- 
cient for many source configurations. Results also indicated that longer dwell times 
for measurements may add value for the more challenging cases. However, it was 
not obvious that “wait-in mode” concepts of operations provide advantages for cer- 
tain threats. 


PHASE III TEST REPORT — CONCLUSION 

The Phase III Test Campaign was a critical piece of a larger effort to evaluate 
the performance of ASP systems. Phase III testing was focused on testing the ASP 
systems with a significantly expanded variety of sources, shielding and “masking” 
configurations, and concepts of operations. The results have provided an additional 
data set in the on-going comparison of ASP performance to current systems. At the 
same time, the data gained from Phase III testing has limits, and it is critical that 
results are interpreted in the context of the original test objectives. 

ASP-IRT REPORT — INTRODUCTION 

The ASP-IRT was tasked with providing two elements of assessment: (1) “the 
testing approach, from contractor testing through operational testing, processes em- 
ployed, specifications, test procedures, and analysis methods;” and (2) “the prob- 
ability of success to detect and identify radiation and nuclear threats and assess the 
performance of the ASP [systems] compared to the first generation systems.” The 
ASP-IRT Report was a culmination of analyses aimed at assessing these two ele- 
ments, conducted from August 2007 through February 2008. These analyses were 
based on information provided by DNDO and CBP, as well as other outside sources. 
This information included DNDO test plans, test reports from several ASP evalua- 
tions, and numerous discussions with officials from both DNDO and CBP. However, 
the analysis and conclusions reached were completely independent of either DNDO 
or CBP, and the resulting conclusions reflected the assessment of the ASP-IRT 
members. 


ASP-IRT REPORT — INITIAL FINDINGS 

The ASP-IRT Report includes an Executive Summary which highlights conclu- 
sions of the document. 

While the ASP-IRT did not concur with assertions that the GAO made in Sep- 
tember 2007 (GAO-07-1247T) that ASP testing in February through September 
2007 “used biased test methods that enhanced the performance of the ASPs,” it did 
agree with other claims, including the fact that tests were “not designed to measure 
the range of ASP system performance.” In addition, the ASP-IRT indicated concern 
that test results and measures of effectiveness were not properly linked to oper- 
ational outcomes, which led to difficulties in developing conclusions from the results. 
Fundamentally, the ASP-IRT asserted that testing to date was “properly character- 
ized as Developmental Test and Evaluation. Independent Operational Testing has 
not been conducted.” 

In evaluating the performance of ASP systems directly, as compared to first gen- 
eration systems, the ASP-IRT focused solely on ASP secondary scanning operations. 
Based on initial independent analysis, the ASP-IRT concluded that for the 13 ob- 
jects used in Phase I testing, using ASP systems “did not affect the probability of 
a missed threat,” when compared to current generation RIIDs. The ASP-IRT stated 
that this conclusion was based on the assumption that all RIID results of “un- 
known” where resolved by CBP Laboratory and Scientific Services (LSS), which pro- 
vides technical support to CBP Officers at POEs. Yet, the ASP-IRT did allow that, 
based on an alternate assumption in which many RIID “unknown” alarms were re- 
solved in the field, “it appears that ASP could substantially reduce the probability 
of entry for nine of the 13 test objects — for most, by at least 20 to 30 percent and 
possibly by 30 to 50 percent.” The ASPIRT was not able to draw any conclusion re- 
garding the affect of ASP for the remaining four test objects. 

In addition, based on first principles calculations, the ASP-IRT asserted that “the 
relative performance of the ASP [systems] and the RIID depends on several factors.” 
The ASP-IRT argued that sample spectra from both systems would indicate com- 
parable performance if a RIID is optimally placed. However, the ASP-IRT also ac- 
knowledged challenges associated with localizing radiation sources within a con- 
tainer, and the likelihood that operators may target the wrong “hot spot” for sec- 
ondary inspection. The ASP-IRT also stated that “ASP performance could be im- 
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proved in all cases by slowing the passage of the truck through the portal, though 
there would he increased costs.” The ASP-IRT noted the benefit of improved consist- 
ency in scanning provided by ASP systems, as compared to RIIDS, especially by “re- 
ducing the impacts of operator inattention, fatigue, and variability of the placement 
of the RIID.” Finally, the ASP-IRT also noted that “substituting the ASP for RIID 
in secondary screening would reduce the number of cases that qualify for referral 
to LSS under the current CBP CONOPS.” 

Additionally, the ASP-IRT made several additional observations based on their 
evaluation of the ASP program. These included a potential need for a more dis- 
ciplined acquisition process to guide large DHS programs, an independent oper- 
ational test and evaluation process, and a more well-defined requirements process 
to ensure that mission needs are properly accounted for in operational require- 
ments. 


ASP-IRT REPORT — DNDO AND CBP RESPONSE TO INITIAL FINDINGS 

DNDO recognizes the thoughtful evaluation that the ASP-IRT provided to the De- 
partment, and values the critiques that were included in the Final Report. Several 
of the concerns that were raised are valid and the Department is taking steps to 
address these issues. 

Unfortunately, in some instances, analysis was limited to information imme- 
diately available, which was not in all cases a complete and accurate representation 
of events. In addition, due to the short time in which the ASP-IRT was tasked to 
produce a final report, subsequent iterations of information exchange that may have 
normally been performed were not feasible. Subsequently, staff from DNDO has met 
with ASP-IRT members, and many of the concerns that are outlined below were 
discussed. In many instances, it was acknowledged that as additional information 
that was provided to the ASP-IRT during their analysis, alternate conclusions 
emerged. In other instances, it appears that the ASP-IRT stands by its initial con- 
clusions. 

Limitation of analysis to secondary scanning operations. — In Section I.C of the Re- 
port the ASP-IRT states that they “sought to determine the extent to which the use 
of the ASP would impact the frequency of nuisance alarms and the probability of 
illicit radioactive materials passing through [POEs] — taking into account other 
equipment with which it might be used, as well as other means of detecting the il- 
licit materials.” However, in the same section, the ASP-IRT explained that they 
solely “focused its analysis on the use of the ASP in the Secondary screening role.” 
This approach to the analysis discounted the economic and time impacts of scanning 
delays due to high nuisance alarm rates in primary scanning. In addition, it also 
discounted the possibility that certain threats may never be referred to secondary 
screening. In the long term, DNDO and CBP expect that the greatest benefits of 
ASP technologies will be in these primary scanning operations, where DNDO testing 
at NYCT has already shown that ASP systems may reduce nuisance alarm rates 
by more than a factor of 10 (1.70 percent for PVT systems and 0.11 percent and 
0.12 percent for two ASP systems). A reduction of secondary referral rates of this 
magnitude, when averaged over the entire volume of cargo containers entering the 
United States annually, would potentially result in hundreds of thousands fewer 
secondary inspections required each year. The savings that the elimination of these 
inspections would have in the efficient processing of trade and manpower resources 
of CBP should not be ignored in what is argued to be a “system-of-systems” anal- 
ysis. 

False dismissal rates in secondary inspections. — In its “system-of-systems” ap- 
proach, the ASP-IRT initially questioned the decision of DNDO to omit LSS anal- 
ysis of RIID data from comparisons of system performance. DNDO has cited evi- 
dence that RIIDs produced “unknown” alarms in up to 60 percent of cases, leading 
to either increased requirements for physical inspections, or the potential for an in- 
advertent release of a threat. The ASP-IRT analysis instead assumed that all of 
these alarms would be sent to LSS for further analysis. The ASP-IRT raised ques- 
tions about the validity of this assumption, based on contrary evidence from oper- 
ational data, which indicated that actual LSS referral rates were less than one- 
tenth of the expected rates, based on evaluations of RIIDs. However, while acknowl- 
edging this discrepancy, the ASP-IRT only asserted that, “If the ASP were to re- 
place the RIID in Secondary screening, it seems likely that some fraction of these 
‘unknown’ cases would be properly resolved as NORM or else referred to LSS for 
resolution. However, based on the available data, we were not able to determine 
what that fraction would be.” 

The reality is that it is not operationally feasible to send all “unknown” alarms 
to LSS for additional analysis. Operational data indicates that only 3,000 alarms 



15 


are sent to LSS annually — far less than the minimum of 40,000 annually predicted 
by the ASP-IRT. DNDO estimates indicate that using ASP systems in secondary 
scanning operations would reduce the number of alarms requiring LSS analysis to 
approximately 3,000^,000 per year, a number manageable with current LSS re- 
sources. More importantly, this would allow all alarms that should be referred to 
LSS to be actually referred to LSS, ensuring that no threats are mistakenly released 
into the Nation under an “unknown” alarm, even when CBP CONOPs are followed. 

First principles calculations of comparative performance. — In addition to providing 
analyses of available test data, the ASP-IRT performed a series of “first principles” 
calculations which attempted to predict the performance of hypothetical ASP sys- 
tems and RIIDs. These calculations focused on the theoretical signal-to-noise ratios 
of the two systems, based upon distances from source to detector, the size of the 
detector, and the time interval of the scan. The ASP-IRT argues that the advan- 
tages provided by the additional detector size of ASP systems is, in some cases, out- 
weighed by the shorter distance and longer scanning intervals provided by RIID sys- 
tems. However, in initial calculations, the ASP-IRT assumed that RIIDs would be 
able to successfully locate the source “hot spot” and a lengthier (1 to 3 minute) scan 
could be focused on that location, with a source to detector distance of 1 foot. The 
incorrect assumption that a RIID would be able to effectively localize and scan any 
source within 1 foot of a detector drastically affected the outcome, and significantly 
reduced the perceived improvements provided by ASP systems. 

The reality is that the height of containers (up to 13.5 feet) and the requirement 
that an operator hold the RIID (limiting effective detector height to 6 to 7 feet) 
make scanning the entire container surface with the RIID difficult. Additional cal- 
culations done by DNDO, and provided to the ASP-IRT, show that for sources lo- 
cated near the center of a loaded container the RIID is approximately as sensitive 
as the ASP but only over a 2-foot radius circle on the surface of the container. Out- 
side of that radius, the sensitivity falls off drastically. This means that a single 
RIID measurement can only effectively scan approximately 2 percent of the area of 
the container. Test data indicate that it is difficult to accurately locate the correct 
“hotspot” at which to place the RIID, which further erodes the effectiveness of the 
unit. ASP systems, unlike RIIDs, stand 14 feet tall, and provide the ability to uni- 
formly scan the entire contents of a container. In addition, this scan is performed 
in 15 seconds, as opposed to the 1 to 3 minutes per “hot spot” measurement by the 
RIID. To effectively scan the “entire” container with a RIID to the same consistency 
as an ASP would take approximately 1 hour, assuming only 1 minute per scan. 
While the ASP-IRT acknowledges some of these challenges in the Report, they also 
propose alternative solutions, such as improved RIID software, or gantry systems 
for consistent scanning of containers. However, these calculations show that ASP is 
the solution that will effectively scan an entire container quickly, because even 
RIIDs with improved software (one recommendation of the ASP-IRT) would still be 
limited in effective detection ranges based on the smaller detector size and prob- 
ability of localization error. 

Other effects that differentiate ASP Systems and RIIDs. — Finally, in addition to 
the issues highlighted above, DNDO has noted several other issues which affect the 
comparison of ASP systems and RIIDs that were not accounted for by the ASP-IRT. 
First, the ASP-IRT Report fails to account for the possibility of multiple “hot spots” 
in a single cargo container. Because CBP protocols require Officers to scan the en- 
tire container and then focus on the regions of highest detected radiation, threat 
materials with lower emissions could be missed and more intensive scans instead 
focused on other “hotter” locations within a container. Again, the ability of ASP sys- 
tems to scan an entire container in a uniform fashion provides the ability to identify 
threats throughout a container, rather than just those that emit the most radiation. 
Second, while the ASP-IRT highlighted the importance of background radiation and 
the confounding effects that it has on radiation identification, their analysis does 
not account for the fact that ASP systems are designed to shield background radi- 
ation from interfering with the detection of sources in the containers being scanned. 
While ASP systems are shielded by one inch of steel on the back and sides of the 
detector, RIIDs have no similar shielding to focus the detection containers. While 
difficult to quantify, this shielding provides measurable improvement in the ASP 
signal-to-noise ratio when compared to RIIDs. While the ASP-IRT acknowledged 
these additional effects, it did not adopt firm positions as to the associated benefits. 

ASP-IRT REPORT — CONCLUSION 

The ASP-IRT Report provides a valuable independent assessment of the ASP pro- 
gram, and will serve as an important source of information in the eventual decision 
to certify ASP systems. However, their analysis was limited in the scope of informa- 
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tion made available, which in some instances may have resulted in conclusions con- 
trary to those that would have been reached if more information were available. 
Since that time, DNDO has met with the ASP-IRT and provided additional informa- 
tion and discussed differences in conclusions reached by each party. In some in- 
stances, DNDO analysis has shown that ASP-IRT conclusions provided unneces- 
sarily limited appreciation for the improvements that ASP systems offer. These 
issues include the ASP-IRT decision to focus solely on ASP improvements to sec- 
ondary scanning operations, assumptions made by the ASP-IRT concerning refer- 
rals of “unknown” secondary alarms to LSS, the probability of localization error on 
first principles analysis of RIID performance, and other issues which differentiate 
ASP systems and RIIDs. DNDO believes that when these issues are considered, ASP 
systems clearly provide an improvement in operational effectiveness when compared 
to current systems. 

DNDO looks forward to continuing to work with CBP and other partners within 
and beyond DHS to improve the Nation’s ability to detect radiological and nuclear 
threats at our ports and borders. DHS is facing an enormous challenge at our ports 
and borders as it struggles to balance the flow of goods and commerce with the need 
to sufficiently scan cargo for radiological or nuclear threats as it enters our Nation. 
The technologies that DNDO is pursuing in the ASP program are a critical compo- 
nent in addressing that challenge. 


CONCLUSION 

I am confident that our plan for the development and evaluation of ASP systems 
is sound. The Phase III test results show promise from ASP systems, and the ASP- 
IRT has provided a valuable assessment of the program to date. 

I welcome and appreciate the committees’ active engagement with this program, 
and look forward to continuing our cooperation as we move forward together. This 
concludes my prepared statement. Chairman Langevin, Ranking Member McCaul, 
and Members of the subcommittee, I thank you for your attention and will be happy 
to answer any questions that you may have. 
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Mr. Langevin. The Chair now recognizes Mr. Thompson for 5 
minutes. 

STATEMENT OF GEORGE E. THOMPSON, DEPUTY DIRECTOR, 
PROGRAMS, HOMELAND SECURITY INSTITUTE 

Mr. Thompson. Mr. Chairman, Representative McCaul and dis- 
tinguished members, thank you for the opportunity to address the 
subcommittee on the subject of the advanced spectroscopic portal. 

My name is George Thompson, and I am the Chair of the ASP 
Independent Review Team or IRT. Our final report was delivered 
to you and your staff last week. 

An independent expert review is a valuable source of advice, but 
only if the experts are truly expert and only if the advice is truly 
objective. So before talking about the findings of our report, I would 
like to take a moment to talk to you about the IRT itself and the 
process that was used to conduct the review. 

Beginning with myself as Chair, my role was to help frame the 
issues, integrate the team’s efforts and contribute substantively. I 
should mention that I am currently a deputy director of the Home- 
land Security Institute, which is an FFRDC or federally Funded 
Research and Development Center. It was established in 2004 
under the Homeland Security Act. Thus, it was the Congress that 
wisely foresaw the need for DHS to have access to an independent 
objective source of expert technical advice on complex Homeland 
Security problems such as the one we are discussing today. 

In September 2007, Mr. Paul Schneider, then Under Secretary 
for Management, asked me to serve as the IRT Chair. The review 
had been underway for several weeks at the time. 

Upon accepting the role, I reviewed the quality of the team mem- 
bers selected by my predecessor. They were, in a word, out- 
standing: Dr. Alan Berman of the Penn State Applied Research 
Lab; Dr. Dennis Slaughter, formerly of the Lawrence Livermore 
Lab; Dr. Peter Vanier of Brookhaven National Laboratory; Dr. Mi- 
chael Wright of Oak Ridge National Lab and Dr. Kaus-Peter Ziock 
of Oak Ridge. All recognized experts in the basic and applied 
science of nuclear detection. 

However, I also recognize the need for experts in acquisition 
management and testing. Therefore, I asked three other distin- 
guished individuals to serve as reviewers: Mr. Thomas Christie, 
former DOD director of Operational Test and Evaluations; Mr. Wil- 
liam Houley, first director of Defense Acquisition Reform; and Dr. 
Marion Williams, formerly chief scientist at the Air Force Test and 
Operational Center or AFOTEC. These and all participants exe- 
cuted strict conflict-of-interest and nondisclosure agreements. 

Mr. Schneider identified the specific questions that he wanted 
the team to answer, but he gave the IRT free rein in answering 
them. He also made it very clear that the team was free to offer 
any other observations we chose. 

From late August to late October the team reviewed over 120 
documents, interviewed dozens of key staff, both internal and ex- 
ternal at DHS, and traveled to four ports of entry to observe both 
first-generation systems and ASP units in operation. At the time 
our goal was to complete the report by mid-November to inform the 
certification decision by the Secretary. 
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However, in early November as we were drafting our report, the 
IRT learned that the Secretary had chosen to defer that decision. 
Nevertheless, Ms. Duke asked us to complete the report, and we 
delivered an interim version on November 19. 

During the remainder of November, December, January and 
early February, the DNDO and CBP reviewed the interim report. 
They discussed its contents with the team, provided some addi- 
tional data and delivered a written response on February 15. The 
IRT carefully considered the additional data and comments, made 
some important revisions and delivered the final report to DHS on 
February 20 . 

With my remaining time, I would like to provide an overview of 
the final report. Our findings are organized around the two ques- 
tions posed by Mr. Schneider last year to assess the ASP testing 
approach and to assess ASP performance. 

Because the Department intended an initial deployment of ASP 
in the so-called secondary screening role and also because of the 
limitations of the test approach, we focused the performance as- 
sessment on secondary screening. 

With respect to the ASP testing approach, the IRT identified sev- 
eral aspects of the overall testing approach that we believe could 
and should be improved. In general, these include a broader char- 
acterization of system performance and a stronger linkage between 
test results and operational outcomes. 

We also looked at the specific test procedures used in 2007. Al- 
though they were not ideal, we did not find evidence that the test 
results had thereby been biased or manipulated. 

Second, ASP performance. We considered both security, mini- 
mizing the chance of a threat entering the United States, and com- 
merce, minimizing unnecessary inspection of innocent cargo. In 
general, we found that the hand-held systems currently used to 
identify radioisotopes in cargo are characterized by wide variations 
in performance. The ASP could, if it performs in the field as in- 
tended — and if appropriate standard operating procedures are de- 
veloped — could substantially reduce these variations in perform- 
ance and thus reduce some key uncertainties in the Nation’s ability 
to counter the threat of nuclear smuggling. 

Finally, the report also offered a number of observations con- 
cerning the need for greater discipline in requirements in test and 
evaluation oversight. 

In closing, I am grateful for the opportunity to be in service. I 
will do my best to answer any questions you might have and I will 
gladly make myself available to you and your staff for more de- 
teiled discussions if you wish. 

Thank you very much. 

Mr. Langevin. Thank you, Mr. Thompson. 

[The statement of Mr. Thompson follows:] 

Prepared Statement of George E. Thompson 
March 5, 2008 

INTRODUCTORY REMARKS 

Mr. Chairman, Representative McCaul, and distinguished Members: Thank you 
for the opportunity to address the subcommittee on the subject of the Advanced 
Spectroscopic Portal. My name is George Thompson, and I am the Chair of the ASP 
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Independent Review Team (IRT). Our Final Report was delivered to you and your 
staff last week. The report is considered For Official Use Only, so I will be providing 
a general overview rather than a detailed description of the team’s findings. 

An independent review is a valuable source of advice for decisionmakers — but only 
if the experts on the review team are truly expert, and only if the advice they pro- 
vide is truly objective. So before I talk about our findings, I’d like to describe the 
IRT itself and process that was used to conduct the review. 

THE INDEPENDENT REVIEW TEAM (IRT) CHAIR 

As Chair of the IRT, my own role was to help frame the issues and integrate the 
contributions of the IRT members into a coherent report. I also contributed sub- 
stantively in those areas in which I am personally knowledgeable. My formal back- 
ground is in Applied Mathematics. I have spent the last 30 years as a practitioner 
of Operations Arialysis, which is really just disciplined problem-solving, using the 
tools of mathematics, probability and statistics, simulation modeling, and systems 
analysis — with a healthy measure of critical thinking and common sense thrown in. 
I am currently a Deputy Director of the Homeland Security Institute. The Institute 
is what is known as a federally Funded Research and Development Center 
(FFRDC). It was established in 2004, pursuant to section 312 of the Homeland Secu- 
rity Act, which specified that the Institute was to be administered by the Science 
and Technology Directorate on behalf of the entire Department of Homeland Secu- 
rity (DHS). Thus, at the same time it established DHS, the Congress wisely foresaw 
the need for the Department to have a knowledgeable, independent and objective 
source of expert technical advice on complex homeland security problems — and that 
is the mission of the Homeland Security Institute. 

In September 2007, Mr. Paul Schneider, then Under Secretary for Management, 
asked me to serve as the IRT Chair. At that time, the review had already been un- 
derway for several weeks. Two other individuals. Dr. Pete Nanos of the Defense 
Threat Reduction Agency, and Mr. John Higbee of the Defense Acquisition Univer- 
sity had, in turn, served briefly in this role but had to withdraw their services. 

THE IRT MEMBERS 

When I accepted the role of IRT Chair, one of my first actions was to review the 
qualifications of the team members that had already been selected by my prede- 
cessor. They were, in a word, outstanding. 

• Dr. Alan Berman of the Penn State Applied Research Lab is a renowned expert 
in signal processing who has served on numerous advisory panels for the U.S. 
Navy and Office of the Secretary of Defense. 

• Dr. Dennis Slaughter, formerly of the Lawrence Livermore National Laboratory, 
is an expert in low energy nuclear physics and cargo security. 

• Dr. Peter Vanier of Brookhaven National Laboratory is an expert in the detec- 
tion of nuclear weapons. He is also a member of the so-called Regional 
Reachback team that analyzes gamma-ray spectra submitted by State and local 
law enforcement organizations. 

• Dr. Michael Wright of Oak Ridge National Laboratory is another Reachback an- 
alyst who is also expert in instrument development and systems integration. 

• Dr. Klaus-Peter Ziock of Oak Ridge is a recognized authority on the subject of 
systematic noise and its impacts on radiation detection. 

These individuals are certainly well-qualified in the basic science of nuclear detec- 
tion. 

However, it was clear to me that the review would also need to consider important 
issues involving acquisition management, systems engineering and the basic prin- 
ciples of test and evaluation. Accordingly, I asked three other individuals with dis- 
tinguished careers in the Department of Defense (DoD) to serve as reviewers of the 
draft report. 

• Mr. Thomas Christie was formerly the DoD Director of Operational Test and 
Evaluation. 

• Mr. William Houley was the first Director of Defense Acquisition Reform and 
a former Director of Test and Evaluation on the staff of the Chief of Naval Op- 
erations. 

• Dr. Marion Williams was formerly Chief Scientist and Technical Director of the 
Air Force Operational Test and Evaluation Center. 

In addition, a small group of technical support analysts — Mr. James Hurd, Mr. 
Bruce Shelton, and Ms. Georganne John — provided valuable assistance in areas 
such as systems engineering, process modeling, and program management. 
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ENSURING OBJECTIVITY AND INDEPENDENCE 

All these individuals — and indeed, all individuals who had access to the study in 
progress — were required to execute strict conflict of interest and nondisclosure 
agreements. As IRT chair, I had full visibility into the team’s deliberations, and at 
no time did I observe anything less than an intellectually honest and open discus- 
sion of the issues. 

The DHS Under Secretary for Management provided the team a Terms of Ref- 
erence memorandum, which spelled out the specific questions to be answered. How- 
ever, the IRT had free reign in answering those questions, and the Under Secretary 
made it clear that the team was free to offer any other observations we saw fit to 
provide. 


REPORT CHRONOLOGY 

During the roughly 2-month period from late August to late October, the team re- 
viewed over 120 documents including test plans, test reports, directives, technical 
reports, briefings, and spreadsheets. (Further details are contained in Section II.B 
of the report, and a complete listing is at Appendix 4.) We conducted interviews and 
technical discussions with key staff from the Domestic Nuclear Detection Office 
(DNDO), Customs and Border Protection (CBP), the Pacific Northwest National 
Laboratory, the National Institute of Standards and Technology, the Department of 
Energy’s National Nuclear Security Administration, and others. (The report lists the 
dates of the key meetings.) Team members traveled to four ports of entry to observe 
both first-generation systems and ASP units in operation. (Again, for full details, 
see section II.B of the report). 

At the time, our goal was to complete the report by mid-November, to inform a 
certification decision by Secretary Chertoff. (As you know, the Fiscal Year 2007 Ap- 
propriations Act contained language requiring the Secretary to certify to the appro- 
priations committees that the ASP represents a “significant increase in operational 
effectiveness” compared to first-generation radiation detection and identification sys- 
tems.) However, in early November, as we were drafting our report, the IRT learned 
that the Secretary had chosen to defer that decision. Nonetheless, Ms. Elaine Duke, 
Deputy Under Secretary for Management, asked the team to complete the report, 
since its findings could still be used — for example, to improve a new round of ASP 
testing. We delivered an interim report on November 19, 2007. 

During the remainder of November, December, January, and early February, 
DNDO and CBP reviewed the interim report. They discussed its contents with the 
team, and provided some additional data. DNDO and CBP delivered a written re- 
sponse on February 15, 2008. (That response is included in the report as Appendix 
8.) The team carefully considered each statement in the response and decided 
whether to make changes as a result. (See Appendix 9 of the report.) In some cases, 
the team agreed with DNDO and CBP, and revised the report accordingly. In other 
cases, we disagreed with a DNDO and CBP statement; however, we could see the 
need do a better job in explaining our ideas. In all cases, we were careful to explain 
why we agreed or disagreed, and what changes (if any) we made as a result. 

We delivered the Final Report to DHS on February 20, 2008. 

SCOPE OF REVIEW — PRIMARY AND SECONDARY SCREENING 

Section I.C of the report describes which topics were studied, which were not, and 
why. It is important to understand that our assessment of ASP performance con- 
centrated on the use of the ASP in the so-called Secondary screening role: Primary 
screening detects the presence of radiation in cargo; Secondary screening identifies 
the isotopes to determine whether or not there is a threat. 

One reason we focused on Secondary screening was DHS’s intent, as of last Fall, 
to make an initial deployment to Secondary in order to gain greater operating expe- 
rience with the ASP. We were charged with informing that decision. Another reason 
is that, in our judgment, the existing test data are insufficient to assess the oper- 
ational impact of using the ASP in the Primary role. Section V.C of the report dis- 
cusses the potential benefits and risks associated with using ASP in the Primary 
role, and the reasons why we believe that additional testing and analysis is needed. 

OVERVIEW OF REPORT 

The report includes a chronology of events associated with the review itself, a de- 
scription of the process used to ensure quality and objectivity, a summary of our 
technical approach, the system-of-systems framework that we developed in order to 
assess the operational significance of improved detection and/or identification capa- 
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bility, and, of course, our independent assessment of the ASP test procedures and 
the test results. 

A series of appendices provides additional technical detail, as well a list of source 
documents, biographies of the team members, and a copy of the Conflict-of-interest/ 
Non-Disclosure Agreement (COI/NDA) form that each of them was required to com- 
plete. As mentioned previously, the DNDO and CBP Response to the interim report 
is included as an appendix, as is the IRT’s assessment of that Response. 

The report findings are organized around the Terms of Reference (TOR). The TOR 
asked the team to do two things: first, assess the ASP testing approach; and second, 
compare the performance of the ASP to first-generation radiation detection and 
identification systems. 

REPORT FINDINGS — ASP TESTING APPROACH 

The IRT identified several aspects of the overall testing approach that we believe 
could and should be improved. In general, these include a broader characterization 
of system performance and a stronger linkage between test results and operational 
outcomes. We developed an operational process flow and proposed scoring schema 
that we believe could help DHS do a better job in assessing the operational impact 
of the ASP. We also looked at the test procedures that were used in 2007. Although 
those procedures were not ideal, we did not find any evidence that the test results 
were thereby biased or manipulated. 

REPORT FINDINGS — ^ASP PERFORMANCE 

In assessing ASP performance, the IRT considered both security (minimizing the 
chance that a threat would be allowed to enter the United States) and commerce 
(minimizing the unnecessary screening and inspection of innocent cargo). We identi- 
fied the key variables and made an independent estimate of ASP impacts on secu- 
rity and commerce based on test data, operating experience with first-generation 
systems, physical first-principles, and other factors. As noted earlier, our assessment 
of performance assumes that the ASP is used in the Secondary screening role, to 
replace the hand-held systems that are currently used. 

In general, we found that the hand-held systems currently used to identify 
radioisotopes in cargo are characterized by wide variations in performance. These 
variations derive from the degree to which these systems rely on the judgment of 
the CBP Officer in adjudicating radiation alarms, the degree to which their perform- 
ance depends on source-detector geometry and the ability to localize the source with- 
in the container, and the degree to which their performance can be degraded by op- 
erator inattention or fatigue. 

The ASP could — if it performs in the field as intended, and if appropriate stand- 
ard operating procedures are developed — substantially reduce these variations in 
performance and thus reduce some key uncertainties in the Nation’s ability to 
counter the threat of nuclear smuggling. 

REPORT FINDINGS — OTHER 

Many of the issues associated with the ASP test program are rooted in a larger 
set of issues having to do with the processes by which DHS manages large and/or 
complex acquisition programs. Accordingly, the IRT also offered a number of obser- 
vations concerning the need for greater discipline in DHS acquisition management, 
requirements, and test and evaluation oversight. 

CONCLUDING REMARKS 

I am grateful for the opportunity to be of service and to help inform important 
decisions on homeland security issues such as nuclear smuggling. I will do my best 
to answer any questions you may have, and I will gladly make myself available to 
you and your staff for more detailed discussions if you wish. I respectfully request 
that my formal statement be submitted for the record. Thank you. 

Mr. Langevin. The Chair now recognizes myself for 5 minutes 
for purpose of questions. 

Mr. Oxford, I would like to begin with you. 

As I stated in my opening statements, several findings in both 
reports caused me concern, and I would like to discuss one that I 
found particularly troubling, and perhaps you can help clarify. The 
unclassified executive summary phase of your report states that 
when it comes to identifying mass sources, PVT system perform- 
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ance appears to be better than ASP systems, because the PVT sys- 
tems are on Naturally Occurring Radioactive Material, or NORM. 
As I understand it, the whole reason for pursuing ASP is because 
PVT makes no distinction between threat material or NORM. So 
the ASP is supposed to be able to distinguish between these two 
types of material. 

However, the report indicates that when a threat object is 
masked by NORM, the ASP is only seeing the masking material 
and not the threat object. This, in effect, makes it no better than 
the PVT. In some cases, it could even be more dangerous because 
if the ASP indicates that threat material masked by NORM is 
NORM only, then we have a false negative which could let dan- 
gerous material in. 

However, it seems to me that because PVT cannot distinguish be- 
tween the two at all, that you have the same problem with that 
system. I don’t understand how you would say that the PVT per- 
formed better, and I will get to explain that in some more detail. 

Mr. Oxford. Thank you, Mr. Chairman. In fact in your opening 
statement, you mentioned the nuances that go on in this business. 
You are right on in that regard. 

PVT will alarm on all NORM. In the report it says that it has 
higher alarm rates, which isn’t necessarily the proper metric, as 
you tried to point out. When it does alarm, again, we have a lot 
of nuisance alarms. That gets sent to secondary. 

The problem in secondary, and if you refer to the IRT report — 
this is where they become closely coupled — you are now dependent 
on this small hand-held device to be able to try to find the threat 
in the middle of the masking sources. 

We have done analysis that shows to actually scan an entire con- 
tainer would take upwards of an hour with a hand-held device to 
be able to localize a potential threat. The hand-held devices will ac- 
tually lock onto the highest output from the container and would 
likely pick up the NORM material regardless of what was embed- 
ded in it. So you are right in saying that if you refer to secondary, 
it doesn’t necessarily get better. 

Now, on the other side with ASP, you have to couple the Phase 
III report along with the Phase I report, where we tested against 
the actual threat basis that I referred to in my opening statement. 
In that case where we looked at masking cases with the size of 
sources that are representative of the actual threat that we are de- 
signing against, we were getting probability of detection and ID, 
both in primary and secondary, of greater than 95 percent. 

So when you see the cases in the Phase III report, what you are 
seeing is a reflection of having now extended the size of the sources 
to, in some cases, a source size of less than approximately 15 per- 
cent of the threat basis, so we actually tried to look at how far we 
can extend the performance of ASP under those conditions. Even 
in those cases, we are getting answers of 50 to 60 percent, prob- 
ability of detection ID for ASP against a much broader threat than 
what you see. So you have to really combine the Phase I and Phase 
III test reports to see the broad range of the outcomes. 

Mr. Langevin. To go back to the original question, does PVT per- 
form better against masked material than ASP? 
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Mr. Oxford. Again, it is not a fair comparison to do PVT and 
ASP. What you have to do is actually distinguish between the ma- 
terial, so you actually have to compare the overall alarm rates in 
primary to the ability in secondary to be able to find whether you 
can identify the threat source or not. 

So in this case you have to compare ASP and primary and sec- 
ondary to a PVT and a hand-held device; and secondary, because 
it is the hand-held device that gives you the identification and sec- 
ondary. If you look at the IRT report, they spent a lot of time look- 
ing at the comparison of ASP and the RIID, the hand-held device 
and secondary. We think there are severe limitations of the hand- 
held device and secondary. So in case if you look at the system-to- 
system comparison, PVT does not outperform ASP. 

Mr. Langevin. In its report, the ASP Independent Review Team 
found that DHS could benefit from an independent operational test 
and evaluation process and organization structure to ensure that 
testing measures the operational performance and reliability of the 
new system. 

Mr. Thompson, what were the factors with ASP testing that led 
to the IRT to make this recommendation? For Director Oxford, does 
the DNDO plan, too, use such an independent process or organiza- 
tion in the ASP testing plan for the next several months? 

Beyond just ASP, do you agree with this general finding, and will 
you use an independent organization to conduct testing on other 
technologies developed by DNDO? 

We will start with Mr. Thompson. 

Mr. Thompson. Thank you, Mr. Chairman. 

The factors that were behind that statement I would characterize 
as general principles or best practices, if you will, of test and eval- 
uation. It is not uncommon in other organizations such as DOD to 
make a careful distinction between development tests conducted by 
the developing entity for the purpose of trying to improve the sys- 
tem and refine the design, make it better, prove that it does what 
it can do, versus operational test, where you put it in the oper- 
ational environment, you let the actual operators operate it. 

Since those are too clearly difference types of tests with two 
clearly different kinds of objectives, most organizations will assign 
two different entities to be responsible for those. 

Mr. Langevin. Thank you. Director Oxford. 

Mr. Oxford. Mr. Chairman, we agree with the recommendations. 

As I noted in my opening statement there are several things in 
that report that we said we were going to adopt. I would also like 
to allow Ms. Duke to address this from the Department’s perspec- 
tive as well, but we are already taking steps to bring in inde- 
pendent testing on behalf of the operational basis Dr. Thompson 
mentioned, where we will essentially rotate the lead responsibil- 
ities over the course of the testing that will take place this year. 

For those purely developmental tests, DNDO will still have the 
lead, although we will have an external oversight function that we 
will review all test plans, assign those test plans and then also re- 
port to the Under Secretary for Management about their adequacy. 

When it gets into the operational testing, DNDO will provide 
mainly a support role as opposed to a lead role in the future be- 
cause we did not have this independent capability. 
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Again, I would like Ms. Duke to have the opportunity to address 
how the Department is going to do this. 

Mr. Langevin. Ms. Duke. 

Ms. Duke. Yes. I also agree with the recommendation, and we 
are clearly looking at how we can build the independent oper- 
ational test and evaluation capability for the Department. 

Under Secretary for Science and Technology Jay Cohen has the 
lead on test and evaluation for the Department, and his executive 
agent is George Ryan, who has about 50 years of experience in the 
tests and evaluation. So we do believe that Mr. Ryan, as I stated 
earlier, will serve in what would be equivalent to the director of 
Operational Test and Evaluation role that DOD has for the contin- 
ued testing of the ASP program. 

Mr. Langevin. Thank you. The Chair now recognizes the Rank- 
ing Member for 5 minutes. 

Mr. McCaul. Thank you, Mr. Chairman. 

Again, thank you for holding this hearing. I think the timing 
couldn’t be more relevant. As we discovered yesterday from a com- 
puter seized from the FARC, a terrorist organization in Latin 
America, revealed that there were many communications with Ven- 
ezuela and Hugo Chavez who is aligned with Iran, discussions 
about nuclear material, discussions about dirty bombs. 

I have always been concerned about that alliance to the Middle 
East in our own hemisphere, and this puts it in our own backyard. 
Coming from Texas I am always concerned, without engaging in a 
lot of hyperbole or irresponsible rhetoric, that the border does face 
some challenges in the post- 9/11 world because that is obviously 
where this stuff is going to come from and cross over into. That, 
I think, demonstrates the need for this. 

I commend all of you for your work in this area because it is so 
important. 

Mr. Thompson, your great work in reviewing this ASP system, 
making these outside independent recommendations is very help- 
ful. 

I tend to agree with you that the ASP when perfected will reduce 
the uncertainty in the current process. I think it will provide a 
more accurate system that will be more efficient. They can help the 
flow of commerce at the border, and yet still enter our ports of 
entry and also, at the same time, provide more accurate readings 
in terms of threat material and to distinguish between real threat 
material and nonthreat materials. 

So, having said all of that, I think we can all — I think I am safe 
in saying this — I think we can all agree on the goal here, and that 
is to provide the best technology possible in this screening process. 
I happen to believe the ASP is a way to go in this. 

I will throw this out to all three of you. What do you see as the 
current deficiencies in the ASP in terms of what needs to be im- 
proved upon? Perhaps more for you, Mr. Oxford, what is your time- 
table here in terms of your last round of testing and the ability to 
get this out? Then how, assuming we do perfect this, how soon can 
this be operational in the field? 

Mr. Oxford. Thank you for those questions. 

First of all, I will use a combination of the charts that are avail- 
able to you and try to speak to this in a logical way. When we did 
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the last round of field evaluation systems, we found what a key — 
what I call operational functionalities that CBP required that 
weren’t quite ready for deployment, and this is what led to the Sec- 
retary’s decision to delay the program. Those functionalities are not 
its ability to distinguish and detect and identify the threat. 

Let me just give you some examples of what this means. Once 
you put these systems into this stream of commerce, as you have 
already mentioned, we can’t allow commerce to stop. We are having 
trouble with what I call system stability with some of the systems, 
where if they powered down they would take hours to reboot, which 
means you are essentially cutting down the traffic in that lane. 

The specification that the current ASP contractors are working 
to is they have got to be able to reboot within 1 minute. If they 
use the natural background readings that were already in the sys- 
tem, they have to be able to reboot within 7 minutes if they want 
to collect new background. 

That is a stability function. It is not performance against the 
threat. It is the operability of the system that CBP requires. In a 
similar way, they want their supervisory computer to be able to 
control four traffic lanes at any port of entry. We focused on one 
lane. This is a broadening of the requirement the CBP asks for. 

The contractors are now building to that. What will be the re- 
sults of that — if you look at the chart that’s in front of you — will 
be the first round of testing that we call systems qualifications test. 
That will be done at the vendor’s location where they will start to 
demonstrate to ourselves, to the independent test office, as well as 
to the Customs and Border Protection, that these functionalities 
have now been built into the system and that they are stable. 

That will allow us to mature down the path of taking these out 
to Pacific Northwest National Laboratory, the CBP and the inde- 
pendent test organization we now use, to make sure, once again in 
a field-like environment, that these systems are ready for fielding. 
That will then be followed by deployment to CBP locations at 
CBP’s choice. That is where DNDO will play a support role as op- 
posed to a lead role. 

Independent of that, we will go back and verify that the perform- 
ance against a threat has not been degraded in any means by the 
system upgrades. We always worry that if you upgrade a system 
that somehow you have lost performance elsewhere. So it is par- 
allel to what CBP and the independent test organization do. We 
will take these back out to Nevada to make sure that they have 
not been degraded against the threat. That should lead us to an 
August timeframe, late fiscal year 2008, to a recommendation for 
the Secretary. 

Mr. McCaul. So if we are fortunate, by August we could have 
the new system? 

Mr. Oxford. We will be able to make an immediate production 
and deployment decision if the Secretary chooses to. We have low- 
rate initial production systems sitting in the warehouse floors with 
our current vendors, and we can immediately provide the upgrades 
to those systems, and those would be the initial deployment units. 

So that would be rather rapid while we go into the production 
buy, which is a 4- to 6-month time period to place the order to 
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begin delivery. So we will be able to field roughly 45 systems with- 
in a very short period of time after the decision is made. 

Mr. McCaul. That is very good. 

Mr. Thompson, do you have any comments? 

Mr. Thompson. Yes, sir. In terms of improvements, I will men- 
tion four things: One area just has to do with the normal things 
you do as you take a technology or a system and you mature it, 
so it has to do with things like does the system operate reliably in 
the field. Those things you test and you prove them out over time. 

The second area has to do with some of the details of how you 
set thresholds for detection and identification. That is a pretty com- 
plicated thing that goes on within the algorithms of these systems. 
I think it is fair to say that DNDO is still doing some learning on 
that, again, as you normally do when you develop a system. 

The third area concerns what I call standard operating proce- 
dures, again, something that you do iteratively when you field a 
new system. It is a marriage of the technology and the procedures 
that you used to implement it. 

So, for example, is 2 miles per hour the right speed for the truck 
to go? What exactly should the investigating officer do when he 
sees a certain alarm condition? Those kinds of things. 

My fourth comment is not about the ASP system or technology 
itself, but on the analysis of test data, and you saw this in the re- 
port. It has to do with how you slice and dice the test data. I think 
there are some things that probably could be done better in this 
round of testing, but again that concerns the analysis of data, not 
the ASP system per se. 

Mr. McCaul. Just for clarification again, Mr. Oxford, do you 
think by August you would have the last round of testing and then, 
if successful, it would take maybe 4 to 5 months of production and 
then all the systems could be upgraded in the field at that point? 

Mr. Oxford. The initial upgrades would actually be for the sys- 
tems that were manufactured under what we call low rate of initial 
production. They have already been manufactured. We have been 
precluded by the appropriations language to doing — to spending 
any acquisition money to upgrade those right now. 

So as soon as we would get a decision from the Secretary, we 
would have the ability to upgrade those systems immediately. That 
is not the 4 to 6 months. The 4 to 6 months would be to place the 
orders and to begin receiving deliveries of the brand-new systems 
on top of the 45 that are sitting in the warehouses now. 

Mr. McCaul. Thank you. 

Mr. Langevin. I thank the Ranking Member. 

The Chair now recognizes the gentleman from New Jersey, Mr. 
Pascrell, for 5 minutes. 

Mr. Pascrell. Thank you, Mr. Chairman. 

Mr. Chairman, the Nuclear Detection Office, their cost-benefit 
analysis, it would seem to me, the report says this, does not justify 
the recent decision to spend $1.2 billion to purchase and deploy the 
ASP technology. 

In particular, the Domestic Nuclear Detection Office used, the re- 
port says, “incomplete and unreliable data to evaluate the cost-ben- 
efit.” 
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Mr. Chairman, we just had an example of what is going on in 
Homeland Security last week when we discussed the fence along 
the south, the southern border. Solution 28, Project 28. We saw 
how money was spent there, and now they have to replace every- 
thing. 

I would hope, because I know the great work that Ms. Duke is 
doing since you have come on board, you have only been there a 
little while, but you are doing a good job and I think you are trying 
to get things in order. But there is a perception, you know, why are 
we — how did we ever get ourselves in the position to using incom- 
plete and unreliable data? 

The enemy is not — I mean the enemy, the enemy is within. We 
can’t get it right. We are spending money in a very foolish way. We 
are wasting the money. 

We know in this situation that we need a — you need a primary, 
and you need static screening. I think that is very obvious, it is 
very clear. You might have to spend more money up front, but you 
either pay now or you pay later, and that is what we are dealing 
with. I am very concerned about this general perception, about 
spending money and then going back and having to do a lot of 
redos again and again. 

So, Dr. Oxford, I would like to ask you a question. We need the 
most sophisticated technology at our ports to ensure that there is 
proper nuclear detection. I think we would agree on that. 

But we must strike a balance between the technology advance- 
ments, and, obviously, what it is going to cost. Those are factors 
that have to be taken into consideration. 

It is my belief that the DNDO should not simply invest in new 
technology — this is my opinion — should not simply invest in new 
technology if it does not make a significant advancement over the 
equipment that is already deployed in the field. If it is not going 
to be a significant improvement, we are wasting money. 

So my question to you is, do Customs and Border Protection 
members, officers in the field, do they regard advanced 
spectroscopic portal monitors as a significant leap forward in detec- 
tion over the equipment that is presently used in the field? 

The follow-up question to both Dr. Thompson and Under Sec- 
retary Duke is, do you have any additional thoughts on this ques- 
tion? 

Let me start with Dr. Oxford. 

Mr. Oxford. Thank you. First of all, I need to set the record 
straight, because this $1.2 billion number is out there, and it is in 
the press, and it is wrong. But it is the total value of the contracts 
we signed. 

But right now, if we live with the deployment decisions that we 
have planned with CBP, the total acquisition of systems would re- 
sult in about 350, not that that is a small number, but about $350 
million. So we have a contract flexibility to buy up to that amount. 

Mr. Pascrell. The total project is $1.2 billion, what we are talk- 
ing about specifically. But we can’t get through the specific project 
unless we go through each of those, so the $1.2 billion 

Mr. Oxford. It is a contract ceiling. Contract ceiling versus what 
we plan to spend. 



29 


Mr. Pascrell. I understand that. I think the Secretary will ulti- 
mately make the decision, not CBP, as to whether this represents 
a significant increase in operational performance. We are not 
spending the acquisition money in the amount that you are citing 
until the Secretary makes that decision, so we have not gone out 
and started to spend that money. 

We are spending development money to get to the point where 
we can make that acquisition decision through the testing program 
that I have played out today. 

CBP is a partner in this. I will tell you if you look at the num- 
bers at Los Angeles/Long Beach, they are getting 400 to 500 nu- 
clear alarms per day. They have dedicated almost 200 officers to 
this mission to help resolve those alarms. Current projections of 
ASP performance would allow that number to come down to 20 to 
25 alarms that they would have to take seriously, which is a tre- 
mendous improvement in their flexibility to manage all the mis- 
sions that CBP 

Mr. Pascrell. Detected in one example, in one specific area we 
are talking about, what have we detected? What have we con- 
cluded? What have we found? Are they all false alarms? Are they 
all relevant? None of them, all of them? 

Mr. Oxford. No. So far if you look at the test data we have 
available, we have been able to against the threat basis that I men- 
tioned in my opening statement that we are getting probability of 
identification and detection against those threats of greater than 
95 percent. That is a much higher number than anything that is 
in the field today. 

Second, when we looked at inserting these systems into an oper- 
ational port, and we did this at the port of New York-New Jersey, 
we are able to show that the secondary referral rates right now 
would come down by at least a factor of 10. So the amount of sec- 
ondary referrals that the operator would have to pay attention to 
is down by at least a factor of 10, maybe as much as 20. 

That is the test data we have available. We will evaluate that 
over the course of the test program that I have laid out for you 
today. 

Mr. Pascrell. Would you agree when I am saying that we need 
a primary and secondary screening process so that we have a 
backup system that works that may be more cost-effective even 
though you have to spend more money up front? 

Mr. Oxford. Absolutely. That is the current configuration at our 
ports, at our land-border crossings. That is the model that we will 
continue to follow and just make CBP more effective. 

Mr. Pascrell. Madam Under Secretary. 

Ms. Duke. I agree. I think the position of CBP which was consid- 
ered in the Secretary’s decision was that they see significant prom- 
ise in the ASP technology, but that the amount of operational test 
and evaluation done introduced too much risk. 

In all the decisions we have to make, Vayl has to make in terms 
of being the program executive, are really risk management. You 
have to tradeoff the risk of schedule, of performance, of putting out 
a technology that is not ready to be done. So throughout this proc- 
ess we will continue to manage that risk so that we can get the 
technology out as quickly as possible if the test results continue to 
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demonstrate it is an improvement in threat detection and identi- 
fication, but not put it out so quickly that the risk is actually in- 
creased because the operators aren’t ready to use it. That is what 
we are going forward with CBP and DNDO. 

Mr. Thompson. Sir, just three comments, if I may. First — and I 
guess this is my own clarification of the record — I believe the lan- 
guage that you have read on incomplete and unreliable data for the 
cost-benefit analysis is actually from a GAO report that was either 
circa October 2006 or 

Mr. Pascrell. I think it is. They did a report also. 

Mr. Thompson. That is correct. 

Mr. Pascrell. You don’t identify with what they said? You don’t 
agree with that? 

Mr. Thompson. We were not tasked to look at the cost-benefit 
analysis or, I should say, to renew the cost-benefit analysis. I will 
say that since the time the cost-benefit analysis was done, there is 
much more test data. If the analysis were to be redone, it would 
use the new and updated data, I am sure. 

The second point I wanted to make is I agree with your state- 
ment that it is important to look at both the costs and the compari- 
son to the existing equipment. Absolutely. That was the reason for 
the second question that Mr. Schneider asked us to compare the 
ASP to the existing fielded systems. 

I won’t quote the numbers here in open session, but I think you 
can see that there are some potentially pretty significant improve- 
ments in terms of reducing the probability of a missed threat, even 
just with a deployment to secondary, assuming that the standard 
operating procedures and so forth can be gotten right. 

The third point I want to make about ASP, the operator, is just 
an antidote to illustrate that that can be a pretty tricky affair. 
When the team went out to Long Beach and they had both the 
first-generation systems and the ASP side by side, and we said to 
the operators how do you like this ASP thing, they said — one of 
them said, I don’t like it. We said, why not? It disagrees with my 
first-generation system. I said, well how do you know the first-gen- 
eration system is telling you the right thing? 

So it is — ^you don’t always get the carefully considered result by 
just asking the operator. 

Mr. Pascrell. Of course the response from the operator is — you 
know, has to be tested too. But it would seem to me that that infor- 
mation must be part of your conclusion. These are the people that 
are going to be using this equipment. Their input into the equip- 
ment, and how the equipment should be used and what they are 
looking for would seem to me to be essential if we are ever going 
to protect the country. 

Mr. Thompson. Absolutely, I agree with you, sir, so the operator 
is the expert when it comes to how this system will be used and 
whether it is useful to him in that way. 

Mr. Pascrell. Well, don’t you ask the people who are going to 
use the equipment: What don’t you have that you should have? 
Don’t you kind of work that into the criteria of what new stuff is 
going to come on line? 

Mr. Thompson. Yes, sir, that is an important step in the process. 

Mr. Pascrell. Thank you, Mr. Chairman. 
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Mr. Langevin. The Chair now recognizes the gentlelady from the 
Virgin Islands, Ms. Christensen, for 5 minutes. 

Mrs. Christensen. Thank you, Mr. Chairman, and thank you for 
holding this hearing. 

Let me just say that the whole thing, as my colleague before me 
said, we had the hearing on the fence and the technology on the 
fence and funding money that was spent, to only now find that we 
have to redo certain parts of the system. It really troubles me that 
a request for appropriation would be made to purchase a tech- 
nology and we would have to say test it first. 

So I am hoping — and I think I heard from Ms. Duke that more 
independent review teams will be set up and a process will be in 
place so that we don’t have to go through this over and over again. 

Some of my questions have to do with the process, and I guess 
I would start with Dr. Thompson. 

One of your findings was that in the Phase I test, the team didn’t 
find a document that laid out the requirements of the operational 
performance requirements and, instead, just a system specification 
that ASP should provide 95 percent or greater probability of detec- 
tion. You concluded that this very reliance on system specification 
versus operational performance requirements made it difficult to 
achieve the real purpose of the test, which was to measure the 
progress of the ASP program toward meeting the operational pro- 
gram objectives. 

Can you expand on that, explain what the concern was there? 

Mr. Thompson. Yes, ma’am, I will certainly try. 

Again, it has to do I think with what questions you ask when you 
are trying to gauge a program like this. 

You can ask the question: Does this machine here have the abil- 
ity to detect such and such of a substance at 95 percent prob- 
abilities, or whatever probability, and that is one question you can 
ask and that’s a good and useful question. 

But there is a larger question as well which is: What is the prob- 
ability that a threat will actually get through at the ports? There 
has to be a way to relate the answer to the first question to the 
answer to the second question, and that was one step that we 
found difficult to do. 

The good news is that the team felt compelled to be as construc- 
tive as possible so we actually constructed an end-to-end analysis 
framework and offered it to the Department as an example of how 
one might take the existing results and use them to answer that 
larger question. 

Mrs. Christensen. Dr. Oxford, have later tests followed that 
have addressed this problem? How would you specify the mission 
objectives, perhaps, in response to this concern from IRT? 

Mr. OxEORD. Yes, ma’am. A couple of points that I would like to 
point out. First of all, we are going back and we are addressing the 
specific requirements of the customer in terms of the functionalities 
they need in the field — as I pointed out earlier — that we will make 
sure that the unique features of these systems that CBP requires 
are built into the system and adequately tested, so that when these 
are delivered they will function as CBP requires. 

The other factors that Dr. Thompson mentions in terms of how 
do we score, how do we connect the operational outcomes was again 
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something — if we go back to the Chairman’s comments, it is very 
nuanced. The fact that we weren’t able to directly explain the test 
data in some cases and how we scored it made it hard for some 
people in the limited amount of time to connect that to the oper- 
ational outcome. 

We accept that, and we are trying to go come up with a better 
scoring mechanism where people can look at the results and then 
can make a direct comparison of that. That is what we will use 
when we go to the Secretary for any decision. 

Mrs. Christensen. The processes that you are setting up for 
independent review and analysis of new technology goes beyond the 
technology related to nuclear material, correct? 

Ms. Duke. Yes. 

Mrs. Christensen. Because several issues have been brought to 
our attention, for example, on the new passport card and changes 
that are being made; the fact that the new one doesn’t have these 
identifying things on the back. So technology like this also — has it 
been going through the same process, being tested for efficiency 
and effectiveness? 

Because it would just seem to me, just as a layperson in this 
field, that having this embedded information would make the card 
a better card. 

Ms. Duke. Yes, there are two pieces to it. One is the test-and- 
evaluation piece, which we are building. We do think we need an 
independent test capability. Whether we rely on one that exists in 
the Department of Defense or somewhere else or build our own has 
not been decided. We do think we need that sustained independent 
operational test capability. 

The second piece is in terms of just as these investments, wheth- 
er it SBInet, whether it is one of our credentialing, our identifica- 
tion cards, what we are building is an investment review process 
where these major programs — and we have about 100 of what we 
consider major programs — would come before the Department at 
key milestone decisions. 

So in the case of ASP, we are at between low-rate initial produc- 
tion where they buy a small number of machines to test them, go 
into the production decision, so that is a system that is going to be 
put in for all our major acquisitions. 

What we are doing right now to kind of, since we are just build- 
ing that capability, is we are looking at all the major programs as 
they stand currently. As I said in my opening statement, we have 
done almost 40, what we are calling quick looks, just looking at the 
existing programs that haven’t had oversight sustained, and as- 
sessing them based on risk and then focusing our attention on the 
programs that appear to have the most acquisition risk. 

Mrs. Christensen. Thank you, Mr. Chairman. 

Mr. Langevin. I thank the gentlelady. 

The gentleman from Texas, Mr. Green is recognized. 

Mr. Green. I thank you, Mr. Chairman, and I thank the Rank- 
ing Member as well for hosting this hearing, and I thank the wit- 
nesses for appearing. 

I have some information indicating that GAO has had some con- 
cerns. Is this true? 
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Mr. Oxford. They have looked at the program several times, and 
we continue to respond to their inquiries, yes, sir. 

Mr. Green. Is it true that they have used terms like “biased 
methods to enhance performance results” having been used? 

Mr. Oxford. They have used those terms, yes. 

Mr. Green. How have you responded to their contention that the 
methods have not been, shall we say, using the best practices for 
methods? 

Mr. Oxford. We clearly have disagreed with that, and we were 
happy to see that the IRT, when they took a fresh look at this, 
found no evidence of biasing of the data or manipulation of the 
data. 

Mr. Green. So, do we still have a dispute between IRT and 
GAO? 

Mr. Oxford. I wouldn’t want to speak on behalf of the GAO. I 
don’t think they have had to even respond to the IRT report at this 
point. 

Mr. Green. The IRT report has gone to GAO? 

Mr. Oxford. Well, it is separate from DNDO, so I would have 
to ask the Chair or Ms. Duke, as to what the dispensation of that 
report is. 

Ms. Duke. I do not know if — I would have to take a get back on 
that. I do not know if the final report has gone to GAO. 

Mr. Green. How long has the report been one that we would 
label as final? 

Ms. Duke. The final report came — is dated February 20 . 

Mr. Green. Of? 

Ms. Duke. Of 2008. 

Mr. Green. 2008. Is there in the process a procedure or require- 
ment that the report eventually will get to GAO? 

Ms. Duke. There is no requirement to distribute the report at all, 
I don’t believe. What we did within a week of receiving it, we 
briefed the Secretary on it and then immediately we released it to 
our authorizing and our appropriation committees, but that was be- 
cause we knew the continued interest. So it was sent. But to my 
knowledge we have not sent it to GAO. 

Mr. Green. If GAO has the concerns regarding GAO not getting 
the reports and whether or not the concerns have been adequately 
dealt with 

Ms. Duke. If GAO would like the report, I would have no objec- 
tion to releasing it to them. 

Mr. Green. Mr. Chairman, is it appropriate for us to in some 
way expedite this process of having GAO review the report? 

Mr. Langevin. Well, I agree with Mr. Green’s question and the 
suggestion that the GAO should look at the report, and I would 
urge the Department of Homeland Security to forward that report 
to GAO so they might take a look at it. 

Ms. Duke. Yes, Mr. Chairman. I don’t have any objection to that. 

Mr. Langevin. Thank you. 

Mr. Green. Thank you, Mr. Chairman. I yield back the balance 
of my time. 

Mr. Langevin. I thank the gentleman. I thank the witnesses for 
the testimony today. It has been very enlightening. I want to close 
by saying I have been a member of the Homeland Security Com- 
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mittee since its inception when it was first a select committee, and 
of course it was right after 9/11 that it was formed. I can remember 
the great concern of the entire country that we could potentially he 
vulnerable to a nuclear attack, being a material — either a weapon 
or material being smuggled in across our borders and we were to- 
tally unprotected. We have come a long way since then. I commend 
the DNDO and Department of Homeland Security for moving as 
aggressively as they have to ensure that we have maximum cov- 
erage of radiation portal monitors at our ports and border cross- 
ings. This is a daunting task. Obviously the work is not yet done, 
but significant progress has been made at least with the deploy- 
ment of these first generation radiation portal monitors. I know we 
are anxious to get to the newest and best technology available. We 
all hope that it will be as soon as possible, but progress has been 
made and we look forward to more of it in the future. 

Thank you all for the great work you are doing for the country. 
I appreciate it, and I do want to thank the witnesses for the valu- 
able testimony, the Members for their questions, and the Members 
of the subcommittee, including myself, may have additional ques- 
tions for the witnesses and would ask that you respond expedi- 
tiously in writing to those questions. 

Hearing no further business, the subcommittee stands adjourned. 

[Whereupon, at 3:41 p.m., the subcommittee was adjourned.] 
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